Issues in Applying Probability Theory to AGI

This paper analyzes the three most common interpretations of probability (frequentist, subjective, and logical) in the context of AGI, and argues that probability theory cannot serve a central role in the theoretical foundation of AGI. 1 Probability and Its Interpretations Probability theory is the most mature formal model for uncertainty representation and calculation, and has been successfully applied to solve many problems in various domains. In particular, it has played a central role in some representative models in AI [1, 2], Cognitive Science [3, 4], as well as AGI [5, 6]. Even so, there are some long-standing issues that have not obtained enough attention. Consequently, probability theory is often misused, and the research results are not properly justified. I have analyzed some related issues previously [7–10], and will approach the problem from a different angle in this paper, by focusing in the interpretation of “probability” with respect to AGI. A probability function P is defined on a sample space Ω by assigning each event A (represented by a subset of Ω) a real number in [0, 1]. P (A) is called “the probability that event A occurs”, and the function is based on axioms P (Ω) = 1 and P (A ∪B) = P (A) + P (B) if A and B are disjoint [11, 12]. As a formal model, probability theory specifies the (valid) calculations within a probability function, without saying much about how to apply the theory to a practical situation. For this purpose, the notion of “probability” needs to be interpreted, by mapping the probability function into a concrete measurement in the domain. The best known interpretations of probability are [13, 14]: Frequentist interpretation. P (A) is the limit of the occurrence frequency of event A in a sequence of observations. Subjective interpretation. P (A) is the degree of belief of a cognitive system on belief A at a certain moment. Logical interpretation. P (A) is the degree of support or confirmation hypothesis A obtains from a body of evidence. One common feature of all of the above interpretations is that P (A) is not an intrinsic property of A, but is relative, with respect to a sequence of observations, a cognitive system at a certain time, or a body of evidence. Without such a “reference point”, P (A) cannot be meaningfully decided. One problem